1. Course overview
2. Introduction to data science
3. Some examples
4. Pipes in R
5. Visualization with ggplot2

When figures and other external sources are shown, the references are included when the origin is known.

You can find all materials at the following location:

The MS Teams environment can be found here. Participants can join the meeting online here

• Name: Gerko Vink
• Identifies as: Dark data scientist
• Studied to be: Psychologist
• Landed in: Statistics
• Enjoys: Putting easter eggs in software
• Pet peeve: Excessively certain Data Scientists
• Married, with 2 kids
• Favorite guitarist: Jimmy Page
• Still has mortgage

Expertise: Missing data theory, statistical programming, computational evaluation

• Description:
  What happened?
• Prediction:
  What will happen?
• Explanation:
  Why did/does something happen?
• Prescription:
  What should we do?

• Exploratory Data Analysis:
  Describing interesting patterns: use graphs, summaries, to understand subgroups, detect anomalies, understand the data
  Examples: boxplot, five-number summary, histograms, missing data plots, …

• Supervised learning:
  Regression: predict continuous labels from other values.
  Examples: linear regression, support vector machines, regression trees, …
  Classification: predict discrete labels from other values.
  Examples: logistic regression, knn, …

image source

How do you think that data analysis relates to:

• “Data analytics”?
• “Data modeling”?
• “Machine learning”?
• “Statistical learning”?
• “Statistics”?
• “Data science”?
• “Data mining”?
• “Knowledge discovery”?

People from different fields (such as statistics, computer science, information science, industry) have different goals and different standard approaches.

• We often use the same techniques.
• We just use different terms to highligh different aspects of so-called data analysis.
• All the terms on the previous slides are not exact synonyms.
• But according to most people they carry the same analytic intentions.

In this course we emphasize on drawing insights that help us understand the data.

Source: wikimedia commons and MIMP summerschool slide 28

Challenger space shuttle - 28 Jan 1986 - 7 deaths

Source: ISLR2, pp. 2

Ginsberg, J., Mohebbi, M., Patel, R. et al. Detecting influenza epidemics using search engine query data. Nature 457, 1012–1014 (2009)

• When high risk decisions are at hand, it paramount to analyze the correct data.

• When thinking about important topics, such as whether to stay in school, it helps to know that more highly educated people tend to earn more, but also that there is no difference for top earners.

• Before John Snow, people thought “miasma” caused cholera and they fought it by airing out the house. It was not clear whether this helped or not, but people thought it must because “miasma” theory said so.

• If we know flu is coming two weeks earlier than usual, that’s just enough time to buy shots for very weak people.

The above examples have in common that data analysis and the accompanying visualizations have yielded insights and solved problems that could not be solved without them.

• On some level, humans do nothing but analyze data;
• They may not do it consistently, understandibly, transparently, or correctly, however;
• Data analyses and visualizations help us process more data, and can keep us honest;
• Data analysis and visualizations can also exacerbate our biases when we are not careful.

Source: Mike Lee

Data wrangling is the process of transforming and mapping data from one “raw” data form into another format. - The process is often iterative - The goal is to add purpose and value to the data in order to maximize the downstream analytical gains

Source: R4DS

• Discovering: The first step of data wrangling is to gain a better understanding of the data: different data is worked and organized in different ways.
• Structuring:The next step is to organize the data. Raw data is typically unorganized and much of it may not be useful for the end product. This step is important for easier computation and analysis in the later steps.
• Cleaning: There are many different forms of cleaning data, for example one form of cleaning data is catching dates formatted in a different way and another form is removing outliers that will skew results and also formatting null values. This step is important in assuring the overall quality of the data.
• Enriching: At this step determine whether or not additional data would benefit the data set that could be easily added.
• Validating: This step is similar to structuring and cleaning. Use repetitive sequences of validation rules to assure data consistency as well as quality and security. An example of a validation rule is confirming the accuracy of fields via cross checking data.
• Publishing: Prepare the data set for use downstream, which could include use for users or software. Be sure to document any steps and logic during wrangling.

Source: Trifacta

boys <- 
  read_csv("boys.txt") %>%
  head()

It effectively replaces head(read_csv("boys.txt")).

Your code becomes more readable:

• data operations are structured from left-to-right and not from in-to-out
• nested function calls are avoided
• local variables and copied objects are avoided
• easy to add steps in the sequence

Benefit: a single object in memory that is easy to interpret

In a %>% b(arg1, arg2, arg3), a will become arg1. With . we can change this.

mice::boys %>% 
  lm(age ~ hgt)

## Error in as.data.frame.default(data): cannot coerce class '"formula"' to a data.frame

VS

mice::boys %>% 
  lm(age ~ hgt, data = .)

## 
## Call:
## lm(formula = age ~ hgt, data = .)
## 
## Coefficients:
## (Intercept)          hgt  
##     -9.7438       0.1443

The . can be used as a placeholder in the pipe.

• We can process a lot of information quickly with our eyes
• Plots give us information about
  • Distribution / shape
  • Irregularities
  • Assumptions
  • Intuitions
• Summary statistics, correlations, parameters, model tests, p-values do not tell the whole story

ALWAYS plot your data!

Source: Anscombe, F. J. (1973). “Graphs in Statistical Analysis”. American Statistician. 27 (1): 17–21.

Source: https://www.autodeskresearch.com/publications/samestats

Layered plotting based on the book The Grammer of Graphics by Leland Wilkinsons.

With ggplot2 you

1. provide the data
2. define how to map variables to aesthetics
3. state which geometric object to display
4. (optional) edit the overall theme of the plot

ggplot2 then takes care of the details

1: Provide the data

boys %>%
  ggplot()

2: map variable to aesthetics

boys %>%
  ggplot(aes(x = age, y = bmi))

3: state which geometric object to display

boys %>%
  ggplot(aes(x = age, y = bmi)) +
  geom_point()

Create the plot

gg <- 
  boys %>%
  ggplot(aes(x = age, y = bmi)) +
  geom_point(col = "dark green")

Add another layer (smooth fit line)

gg <- gg + 
  geom_smooth(col = "dark blue")

Give it some labels and a nice look

gg <- gg + 
  labs(x = "Age", y = "BMI", title = "BMI trend for boys") +
  theme_minimal()

plot(gg)

source